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DIAGNOSTIC ALGORITHM BENCHMARKING 

Scott Poll (NASA Ames Research Center) 


Objectives 

Benchmark diagnostic algorithms 
(DAs) using standardized platform 
Compare performance empirically 
Facilitate research in and 
maturation of diagnostic 
technologies 


Challenges 

Various diagnostic approaches 
(expert systems, model-based, 
data-driven, stochastic) 
Diagnostic algorithms support 
different operational contexts - 
difficult to define evaluation 
criteria 


Approach 

Acquire nominal and faulty experimental 
data with known ground truth 
Use standard formats for system 
description, data, and diagnosis results 
Create software framework to execute 
diagnostic algorithms and evaluate 
performance 


Diagnostic Framework (DXF) 


Implementation 





DXC’10 Diagnostic Problems 


ADAPT-Lite 


24V DC > 


High-level representation of physical 
system description, sensor data, 
diagnosis output 

Run-time architecture for executing DAs 
with experimental scenarios 
Evaluation component that evaluates DAs 
using pre-defined metrics 



Aspect 


system 


operational scenario 


diagnostic use case 


#comps 


ffmodes 


initial relay state 


initial circuit breaker 
state 


nominal mode changes 


multiple faults 


offset 


drift 

(incipient) 


intermittent 

offset 



DP-1 


ADAPT-Lite 
single-string 
UAS mission 

abort rec. 

25 

102 

closed 

closed 

no 

no 

yes 

yes 

yes 


ADAPT 
redundant 
systems UAS 
mission 
fault 

recovery rec. 

96 

306 

open 

closed 

yes 

yes 

yes 

no 

no 


ADAPT 

Two system descriptions created from the 
ADAPT Electrical Power System testbed 
Archived ~4 minute nominal and faulty 
scenarios with known ground truth for 
ADAPT-Lite and ADAPT systems 


DXF and ADAPT EPS scenarios 
used in two diagnostic competitions 
(DXC’09, DXC’10), hosted by the 
International Workshop on 
Principles of Diagnosis 
DXC’10 introduced new challenges: 
new fault types, reduced sensor 
set, multiple sample rates 


Results (only DXC’10 DP-1 shown, see links for more information) 
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DP-1 cost breakdown by scenario fault type 


DP-1 classification error by scenario detection type 
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'mission 


DA 

Cost 

Rank 

Adapted FACT 

2250 

2 

HyDE-A 

6950 

6 

ProADAPT 

4925 

5 

QED 

2350 

3 

SystemicsC 

2400 

4 

TARDEC 

2000 

1 


By comparison, 

DA that always aborts = 2225, 
DA that never aborts = 8125, 
Minimum possible cost = 0 


Metric 

Name 

Category 

M fd 

fault detection time 

detection 

M fn 

false negative rate 

detection 

MM 

false positive rate 

detection 

M da 

scenario detection accuracy 

detection 

M fi 

fault isolation time 

isolation 

M err 

classification errors 

isolation 

1 EH 

CPU load 

computation 

M mem 

memory load 

computation 


V, 

DP-1 classification error by scenario fault type 

No DA dominates all metrics 
Real-world system noise, 
latencies, transients, and 
coding errors resulted in DA 
false positives and 
classification errors 


Publications and Data Sets 

ADAPT Electrical Power System information, software 
framework, sample data, test data, results, publications 
and presentations are available on DASHIink: 

• DXC’09: https://c3.ndc.nasa.gov/dashlink/proiects/36/ 

• DXC’10: https://c3.ndc.nasa.gov/dashlink/proiects/33/ 

Team: Scott Poll (NASA Ames), Sriram Narasimhan (UARC @ NASA Ames), Tolga Kurtoglu (PARC), David Garcia (PARC), Johan de Kleer (PARC), Alexander Feldman (Delft 
University of Technology & PARC), Arjan van Gemund (Delft University of Technology) 

NASAAviation Safety Annual Technical Meeting, St. Louis, MO May 10 - 12, 2011 


DA 

M h (s) 


jyus) 


M rn..(ms) 

M mpm (kb) 

Adapted FACT 

21.462 

0.069 

0.040 

0.901 

151.746 

98.000 

37189 

9656 

HyDE-A 

27.717 

0.873 

0.000 

0.240 

29.355 

136.030 

1550 

6463 

ProADAPT 

15.990 

0.179 

0.019 

0.825 

64.711 

171.000 

6356 

4373 

QED 

7.307 

0.015 

0.105 

0.882 

115.499 

71.752 

239 

5364 

SystemicsC 

9.390 

0.134 

0.026 

0.856 

13.860 

73.000 

229057 

3151 

TARDEC 

162.638 

0.090 

0.000 

0.922 

162.638 

58.000 

8979 

3211 

























































